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Abstract 

The reliability function of variable-rate Slepian-Wolf coding is linked to the reliability function of channel coding 
with constant composition codes, through which computable lower and upper bounds are derived. The bounds coincide 
at rates close to the Slepian-Wolf limit, yielding a complete characterization of the reliability function in that rate 
regime. It is shown that variable-rate Slepian-Wolf codes can significantly outperform fixed-rate Slepian-Wolf codes 
in terms of rate-error tradeoff. The reliability function of variable-rate Slepian-Wolf coding with rate below the 
Slepian-Wolf limit is determined. In sharp contrast with fixed-rate Slepian-Wolf codes for which the correct decoding 
probability decays to zero exponentially fast if the rate is below the Slepian-Wolf limit, the correct decoding probability 
of variable-rate Slepian-Wolf codes can be bounded away from zero. 

Index Terms 

Channel coding, duality, reliability function, Slepian-Wolf coding. 


I. Introduction 

Consider the problem (see Fig. 1) of compressing X" = {Xi,X 2 , - ■ ■ ,X„) with side information F” = 
(yi,F 2 ,‘’‘ : F„) available only at the decoder. Here is a joint memoryless source with zero-order 

joint probability distribution Pxy on finite alphabet A x V- Let Px and Py be the marginal probability distributions 
of X and Y induced by the joint probability distribution Pxy- Without loss of generality, we shall assume 
Px{x) > 0,Py(j/) > 0 for all a; G G y. This problem was first studied by Slepian and Wolf in their 
landmark paper HI. They proved a surprising result that the minimum rate for reconstructing X” at the decoder 
with asymptotically zero error probability (as block length n goes to infinity) is H{X\Y), which is the same as 
the case where the side information F" is also available at the encoder. The fundamental limit H{X\Y) is often 
referred to as the Slepian-Wolf limit. We shall assume H{X\Y) > 0 throughout this paper. 

Different from conventional lossless source coding, where most effort has been devoted to variable-rate coding 
schemes, research on Slepian-Wolf coding has almost exclusively focused on fixed-rate codes. This phenomenon 
can be partly explained by the influence of channel coding. It is well known that there is an intimate connection 
between channel coding and Slepian-Wolf coding. Intuitively, one may view F" as the channel output generated 
by channel input X" through discrete memoryless channel Py\x, where Py\x L the probability transition matrix 
from A to F induced by the joint probability probability distribution Pxy- Since F” is not available at the encoder, 
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Fig. 1. Slepian-Wolf coding 


Slepian-Wolf coding is, in a certain sense, similar to channel coding without feedback. In a channel coding system, 
there is little incentive to use variable-rate coding schemes if no feedback link exists from the receiver to the 
transmitter. Therefore, it seems justifiable to focus on fixed-rate codes in Slepian-Wolf coding. 

This viewpoint turns out to be misleading. We shall show that variable-rate Slepian-Wolf codes can significantly 
outperform fixed-rate codes in terms of rate-error tradeoff. Specifically, it is revealed that variable-rate Slepian-Wolf 
codes can beat the sphere-packing bound for fixed-rate Slepian-Wolf codes at rates close to the Slepian-Wolf limij^. 
It is known that the correct decoding probability of fixed-rate Slepian-Wolf codes decays to zero exponentially fast 
if the rate is below the Slepian-Wolf limit. Somewhat surprisingly, the decoding error probability of variable-rate 
Slepian-Wolf codes can be bounded away from one even when they are operated below the Slepian-Wolf limit, 
and the performance degrades graciously as the rate goes to zero. Therefore, variable-rate Slepian-Wolf coding is 
considerably more robust. 

The rest of this paper is organized as follows. In Section [III we review the existing bounds on the reliability 
function of fixed-rate Slepian-Wolf coding, and point out the intimate connections with their counterparts in channel 
coding. In Section Hill we characterize the reliability function of variable-rate Slepian-Wolf coding by leveraging 
the reliability function of channel coding with constant composition codes. Computable lower and upper bounds are 
derived. The bounds coincide at rates close to the Slepian-Wolf limit. The correct decoding probability of variable- 
rate Slepian-Wolf coding with rate below the Slepian-Wolf limit is studied in Section |IV] An illustrative example is 
given in Section |Vl We conclude the paper in Section |Vll Throughout this paper, we assume the logarithm function 
is to base e unless specified otherwise. 

II. Fixed-Rate Slepian-Wole Coding and Channel Coding 

To facilitate the comparisons between the performances of fixed-rate Slepian-Wolf coding and variable-rate coding, 
we shall briefly review the existing bounds on the reliability function of fixed-rate Slepian-Wolf coding. It turns 
out that a most instructive way is to first consider their counterparts in channel coding. The reason is two-fold. 

'Note that the same conclusion is trivially true if the rate is greater than H{X) since in this case one can achieve zero error probability 
using variable-rate coding schemes. 
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First, it provides the setup to introduce several important definitions. Second and more important, it will be clear 
that the reliability function of hxed-rate Slepian-Wolf coding is closely related to that of channel coding; indeed, 
such a connection will be further explored in the context of variable-rate Slepian-Wolf coding. 

For any probability distributions P,Q on X and probability transition matrices V,W : X y, we use H{P), 
I{P, V), D{Q\\P), and D{W\\V\P) to denote the standard entropy, mutual information, divergence, and conditional 
divergence functions; specifically, we have 

H{P) = -Y^ P{x) log P{x ), 

X 

D{Q\\P) = Y^Q{x)\og^ 

D{W\\V\P) = ^ P{x)W{y\x) log 

The main technical tool we need is the method of types. First, we shall quote a few basic definitions from ||2l. 
Let V{X) denote the set of all probability distributions on X. The type of a sequence a;" G X^, denoted as P^n, is 
the empirical probability distribution of x”. Let Vn{X) denote the set consisting of the possible types of sequences 
x" G X". For any P G Pn(X), the type class Tn{P) is the set of sequences in X^ of type P. We will make 
frequent use of the following elementary results: 


\Vn{X)\ < (n + l)l-^l, 

n 

[|p(x,) = x" e r„(Q),Q G Pn{X),PG V{X). 

i=l 

A block codq3 Cn is a set of sequences in A"”. The rate of C„ is defined as 


( 1 ) 

( 2 ) 

( 3 ) 


P(C„) = i logical. 

n 

Given a channel lLV|x ■ X ^ y, a block code C„ C A””, and channel output G 3^", the output of the optimal 
maximum likelihood (ML) decoder is 

n 

= arg^mm - ^ log WY\xiX,\xi), 

“ ” i=l 

where the ties are broken in an arbitrary manner. The average decoding error probability of block code C„ over 
channel lLV|x is defined as 

Pe{Cn,WY\x) = Pr{X" ^ x"|x"' is transmitted}. 

a:"GC„ 


^More precisely, a block code Cn is an ordered collection of sequences in We allow Cn to contain identical sequences. Moreover, for 
any set A C we say Cn C An if E ^ for all x‘^ E Cn- Note that Cn ^ A does not imply \Cn\ < \A\. 
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The maximum decoding error probability of block code over channel WV|x is defined as 
Pe,ma.x{Cn, Wy\x) = max Pr{X" ^ x^\x^ is transmitted}. 

x'^^Cn 

The average correct decoding probability of block code over channel Wyjx is defined as 

Fo(C„,Wrix) = 1 - Pe(Cn,Wrix). 

Definition 1: Given a channel kfV|x : -T —?> we say an error exponent E >0 is achievable with block codes 
at rate R if for any 5 > 0, there exists a sequence of block codes codes {C„} such that 

liminf i?(C„) > R — S, 

n—^oo 

\imsup--logPe{Cn,WY\x)>E-S- (4) 

n—fcx) TT' 

The largest achievable error exponent at rate R is denoted by E{Wy\x, R)- The function E{Wy\x, ') is referred 
to as the reliability function of channel Wy\x- Similarly, we say a correct decoding exponent E'^ > 0 is achievable 
with block channel codes at rate R if for any i5 > 0, there exists a sequence of block codes {C„} such that 

liminf i?(C„) > R — S, 

n—^oo 

liminf--logPc(Cn,WV|x) ^E^" + 6. 

n—^oo ji 

The smallest achievable correct decoding exponent at rate R is denoted by E'^{Wy\xiR)- It will be seen that 
E^{Wy\x^R) is positive if and only if i? > C{Wy\x)^ where C'(tTVlx) — I{Qx,Wy\x) is the capacity 

of channel WV|x- Therefore, we shall refer to the function E'^{Wy\x, •) as the reliability function of channel WV|x 
above the capacity. 

Remark; Given any block code C„ of average decoding error probability Pe(CrnWV|x)^ we can expurgate the 
worst half of the codewords so that the maximum decoding error probability of the resulting code is bounded above 
by 2Pe(Cn, ffV|x)- Therefore, the reliability function E{Wy\xt‘) is unaffected if we replace Pe(Cn,ITV|x) by 
Te,niax (C„, Wy\x) in ®- 

Definition 2: Given a probability distribution Qx £ P{X) and a channel lTV|x : -T —we say an error 
exponent E > 0 is achievable at rate R with constant composition codes of type approximately Qx if for any 
<5 > 0, there exists a sequence of block codes codes {Cn} with C„ C Tn{Pn) for some £ P„(T’) such that 

lim II -Qx\\ = 0, 

n—¥oo 

liminf R{Cn) > R — 6, 

n—¥oo 

limsup--logPe(C„,lTV|x) > E - S, 

n—fcx) ^ 

where || • || is the li norm. The largest achievable error exponent at rate R for constant composition codes of type 
approximately Qx is denoted by E{Qx, Wy\xi R)- The function E{Qx, iTV|Xi •) is referred to as the reliability 
function of channel lTV|x for constant composition codes of type approximately Qx- Similarly, we say a correct 
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decoding exponent E'^ > 0 is achievable at rate R with constant composition codes of type approximately Qx if 
for any i5 > 0, there exists a sequence of block codes {C„} with C„ C 7^(P„) for some G P„(<Y) such that 


lim \\Pn - <5x11 = 0, 

n—foo 

lim inf R{Cn) > R — S, 

n—^oo 

\\v[i\ni--\ogPc{Cn,WY\x) < + 5. (5) 

n—foo 77 

The smallest achievable correct decoding exponent at rate R for constant composition codes of type approximately 
Qx is denoted by E‘^{Qx,Wy\x,R)- 

Remark: The reliability function E{Qx, Wy\x, ') is unaffected if we replace Pe{Cn, Wy\x) by Pe,max(Cn, Wy\x) 

in (|5]l. 

Let |f|+ = max{0,f} and (x,x) = -logX^y \/WY\x{y\x)WY\x{y\x). Define 


Eex{Qx^WY\X, R) 


QX\X '^Xx\x'^ — ^ - 


Erc{Qx,WY\x,R) = min [D{Vy\x\\Wy\x\Qx) + \I{Qx,Vy\x) - R\'^] 

^Y\X 


Esp{Qx,Wy\x,R) = min 

Vy\x-I{Qx,Vy\x)<R 


^Qxx^^Ylx + I{Qx,Qx\x) ~ ^ 
\x\Qx) + \IiQx, Vy 
D{Vyix\\Wy\x\Qx) 


(6) 

(7) 

( 8 ) 


where in ©, Qjf and Qxx respectively the marginal probability distribution of X and the joint probability 
distribution of X and X induced by Qx and Qx\x- 

Let R'^{Qx, Wy\x) be the smallest i? > 0 with Ef.x{QxiWY\xiR) < oo. We have 


RTx{Qx,Wy\x) = min _ I{Qx,Qx\x)- 

Qx\x -Qx—Qx^^Q^X^^YIX 

It is known ||2l Excercise 5.18] that Eex{Qx, Wy\Xj R) is a decreasing convex function of R for R > Rrx{Qx,WYixy, 
moreover, the minimum in @ is achieved at Qxx 


„ , cQ{x)Q{x) if < oo, 

Qxxix,x) = < 

I 0 otherwise, 

where the probability distribution Q and the constant c are uniquely determined by the condition Qx = Qx- 
It is shown in HU Lemma 3] that, for some R*(Qx,Wy\x) G [0, 7(Qxj we have 


max {Ee,x{QxiWY\XiR)y ErdQx, Py\x, 7?)} 
It is also known |l2l Corollary 5.4] that 


Eex{Qx,WYlX,R) if R<R*{Qx,Wyix), 
ErciQx.WYix,R) if R> R*{Qx,Wyix)- 


ErciQx, WyIX, R) — 


( 11 ) 


EspiQx,WY\Xj R) ^f R> Rcr{Qx,WY\x)j 

EspiQx I Wy\X^ Rcr) + Rcr — R if 0 < P < Rcr{Q X , Wy\x)^ 
where R^r = RcriQx, Wy\x) is the smallest R at which the convex curve Esp{Qx, Wy\xj R) meets its supporting 
line of slope -1. It is obvious that Rcr(Qx,WY\x) < I{Qx,Wy\x)- 
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Proposition 1: Rcr{Qx,WY\x) = I{Qx,Wy\x) if and only if for all x,y such that Qx{x)WY\x{y\x) > 0, 
the value of 

, WY\x{y\x) 

Ex' Qx{x')WY\x{y\x') 

does not depend on y. 

Proof: See Appendix lAl ■ 

Define ffV|x) = inf{-R > 0 : Esp{QxjWy\xtR) < oo}- ft is known ^ Excercise 5.3] that 


R%{Qx,Wy\x) = min/(Qx, 


( 12 ) 


where the minimum is taken over those VV|x’s for which VY\x{y\x) = 0 whenever WY\x{y\x) = 0; in particular, 
R%{.Qx,Wy\x) > 0 if and only if for every y Gy there exists an x G with Qx{x) > 0 and WY\x{y\x) = 0. 

Proposition 2: The minimum in ([12]) is achieved at Vyix = kfV|x if and only if the value of 

Wrix(ylx) 

Ex' Qx(x')WYix(ylx') 
does not depend on y for all x,y such that Qx{x)WY\x{y\x) > 0. 

Proof: The proof is similar to that of Proposition (Tj The details are omitted. ■ 

One can readily prove the following result by combining Propositions [T] and |2| 

Proposition 3: The following statements are equivalent; 

1 ) RcriQx, Py\x) = IiQx,WY\x)', 

2) R^piQx^PYix) = HQx, W^vix); 

3) for all x,y such that Qx{x)WY\x{y\x) > 0, the value of 

. WY\x{y\x) 

Ex' Qx{x')WY\x{y\x') 

does not depend on y. 

Proposition 4: 1) E[Qx,Wy\x,R) >xcLa.x{Eex{Qx,WY\x,R),Ej.c{Qx,WY\x,R)Y 

2) E{Qx,Py\x^R) < Esp{Qx,Wy\x^R) with the possible exception of i? = R'^{Qx,Wy\x) at which point 
the inequality not necessary holds; 

3) £;=(Qx,fEr|x,i?) =minw,^ [D{Vy\x\\Wy\x\Qx) + \R - I{Qx-.Vy\xT]- 

Remark: i?ex(Qx, WV|x, f?). Erc{Qx,WY\x,R)^ and Esp{Qx,Wy\x,R) are respectively the expurgated expo¬ 
nent, the random coding exponent, and the sphere packing exponent of channel Wfix for constant composition 
codes of type approximately Qx- The results in Proposition |4| are well known. However, bounding the decoding 
error probability of constant composition codes often serves as an intermediate step in characterizing the reliability 
function for general block codes; as a consequence, the reliability function for constant composition codes is rarely 
explicitly defined. Moreover, Eex{Qx, Wy\x,R)^ Erc{Qx,WY\XT R), and Esp{Qx,Wy\Xt R) are commonly used 
to bound the decoding error probability of constant composition codes for a fixed block length n; therefore, it is 
implicitly assumed that Qx is taken from (see, e.g., ||2l). In contrast, we consider a sequence of constant 

composition codes with block length increasing to infinity and type converging to Qx for some Qx G V{X) 
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(see Definition |2]l. A continuity argument is required for passing Qx from P„(A’) to V{X). For completeness, 
we supply the proof in Appendix iBl Note that different from E{Qx, Wy\xi ■)> '^he function E‘^{Qx, Wy\xi ') has 
been completely characterized. 

Proposition 5: 1) E{Wy\x, R) = supQ^ E{Qx,Wy\x,R), 

2) E-iWY\x,R) E^{Qx,Wyix,R)- 

Remark; In view of the fact that E‘^{Qx,Wy\xjR) is a continuous function of Qx, we can replace “inf” with 
“min” in the above equation, i.e., 

E‘^{Wyix, R) = m^inE‘={Qx,WY\x, R)- (13) 

Qx 

Proof: It is obvious that E{Wy\x,R) > supg^ E{Qx,Wy\x,R)', the other direction follows from the fact 
that every block code contains a constant composition code with Pe,max{Cn, Wy\x) < Pe,max(Cn, Wy\x) 
and R{C'^) > R{Cn) — \X\ t°s(»+i) ^ similarly, it is clear that E‘^(Wy\x, R) < infgx E^^iQxjWxix, R)', the other 
direction follows from the fact that given any block code C„, one can construct a constant composition code C'^ 
with Pe(c;, Wy\x) <{n + Wy\x) and R{C'„) = R{Cn) H). ■ 

The expurgated exponent, random coding exponent, and sphere packing exponent of channel Wy\x for general 
block codes are defined as follows; 

1) expurgated exponent 

Eex{WY\X,R) = inaxi?ea;((5x, R) , (14) 

Qx 

2) random coding exponent 

Erc{WY\X, R) = ^&^Erc{Qx,WY\X, R), (15) 

Qx 

3) sphere packing exponent 

EspiWY\x, R) = ^b.xEsp{Qx,Wy\x, R)- (16) 

Qx 

Let i?“(tLV|x) be the smallest R to the right of which Esp{Wy\x, R) is finite. It is known ||2] Excercise 5.3] 
Il5l that 


RTp{Wy\x) = maxi?“(Qx,lLy|x) 

wx 


= -log 


min max 
Qx y 


E 

xeX:WY\x(y\x)>o 


Qx{x) 


By Propositions |4] and 13 we recover the following well-known result ||2l. Il5l; 


ma.x{Eex{WY\X, R), Erc{WY\X, R)} < E{Wy\X, R) < Esp{Wy\X, R) 


(17) 


with the possible exception of i? = i?“(lkV|x) at which point the second inequality in (fTTl i not necessarily holds. 
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Now we proceed to review the results on the reliability function of fixed-rate Slepian-Wolf coding. A fixed-rate 
Slepian-Wolf code 0„(-) is a mapping from A”" to a set An- The rate of (/>n(-) is defined as 

= -log|yf„|. 
n 

Given and the output of the optimal maximum a posteriori (MAP) decoder is 

n 

X" = arg min - V logPx|y(a;*]!;) 

I—I 
n 

= arg min logPxY{x^,Yi), 

a:":0„(a:")=0„(X") -f—' 

t—L 

where the ties are broken in an arbitrary manner. The decoding error probability of Slepian-Wolf code (j>n{') is 
defined as 

Pe{^n,PxY)=Pr{X'^ 

The correct decoding probability of Slepian-Wolf code (j)n{‘) is defined as 

Pc{4>n,PxY) = 1 — Pe{4>n, Pxy)- 

Definition 3: Given a joint probability distribution Pxy, we say an error exponent E > 0 is achievable with 
fixed-rate Slepian-Wolf codes at rate R if for any (5 > 0, there exists a sequence of fixed-rate Slepian-Wolf codes 
{4>n} such that 

lim sup R{4>n) < R + S, 

n—^oc 

\imsup--log Peifin, Pxy) > E - 6. 

n—voo ri 

The largest achievable error exponent at rate R is denoted by Ef{PxY, R)- The function Ef{PxY, •) is referred to 
as the reliability function of fixed-rate Slepian-Wolf coding. Similarly, we say a correct decoding exponent E‘^ > 0 
is achievable with fixed-rate Slepian-Wolf codes at rate R if for any (5 > 0, there exists a sequence of fixed-rate 
Slepian-Wolf codes {fin} such that 

lim sup R{fin) < R + S, 

n—^oo 

limini--logPcifin, Pxy) < E^" + S. 

n—^oo TL 

The smallest achievable correct decoding exponent at rate R is denoted by Ej{Pxy,R)- It will be seen that 
Ej:{Pxy, R) is positive if and only if i? < H{X\Y). Therefore, we shall refer to the function Ej{PxY, ■) as the 
reliability function of fixed-rate Slepian-Wolf coding below the Slepian-Wolf limit. 

Fixed-rate Slepian-Wolf coding has been studied extensively 0, Ha-in. The expurgated exponent, random 
coding scheme, and sphere packing exponent of fixed-rate Slepian-Wolf coding are defined as follows; 

1) expurgated exponent 

EfMPxY,R) = min [D{Qx\\Px) + Ee4Qx,PY\x,HiQx) - i?)] , (18) 

Qx 
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2) random coding exponent 

Ef^rciPxY,R)=Tnm[D{Qx\\Px)+ErciQx,PY\x.H{Qx)-R)] , (19) 

Qx 

3) sphere packing exponent 


Ef,sp{PxY, R) = min [D{Qx\\Px) + Esp{Qx, Py\x, H(Qx) — -R)] ■ 

Qx 


( 20 ) 


Equivalently, the random coding exponent and sphere packing exponent of fixed-rate Slepian-Wolf coding can be 
written as 0: 

i+p 


Erc{PxY,R) = max ^ -logT^ 

0<p<l 1 


y X 


'^PxY{x,y) i+f 


EspiPxY, R) = sup < - log V 

p>0 ' 


'^PxYix,y) i+f 


i-Hp 


+ pR /) 


pi? 


To see the connection between the random coding exponent and the sphere packing exponent, we shall write them 
in the following parametric forms 0: 

R = H{X^P^\Y^P^), 

Ef,sp{PxY, R) = D{Px(.p)ym\\Pxy), 


and 


Erc{PxY, R) 


D{Px(p)y(p)\\Pxy) 


-logE 

V 


1 2 


E \/PxY(x,y) 


H" R 


if H{X\Y)<R<H{Xp\Yp)\^^^, 
if R> H{X^E\y(p))\^^^ , 


where the joint distribution of {X^P\Y^P'i) is Px(p)ym^ which is specified by 

r 1 11+P 

Priy) Y.a:Px\Y{x\y)^+» 

PYiP'tiv) = -^' 


Hy’PYiy') Y.xPx\Y{x\y')^+P 


1 H-P ’ 


y&y, 


Ex(p)|y(p) {x\y) — 


Px\Y{x\y) i+p 

Y.X' Px\Y{x'\y)^p 


X G X,y Gy. 


( 21 ) 


( 22 ) 


Define the critical rate 


RfMPxY)= H{X‘^p^Y^p'>) 

p=i 

Note that Erc{PxY, R) and Esp{PxY,R) coincide when R G [ii(X|y), Rf^cr{PxY)]. Let Rfsp{PxY) = sup{i? : 
Ef^sp{PxY,R) < oo}. It is shown in ifTol that 

Rtsp{PxY) = maxlog \{xGX : Px\Y{x\y) > 0}|. 

It is well known that the reliability function Ef{PxYY) is upper-bounded by Ef^sp{PxY,-) and lower-bounded 
by Ef^rc{PxY,-) and Ef^ex{PxY,-) 0^ 0. i-e-> 


ma,x{Ef^rc{PxY,R),Ef^ex{PxY,R)} < Ef{PxY,R) < Ef^sp{PxY, R) 


(23) 
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with the possible exception of R = RYspi^xv) at which point the second inequality in ( l2?t not necessarily holds. 
Note that Ef{PxY,R) is completely characterized for R G [H{X\Y), Rf^criPxv)]- 

Unlike Ef{PxY,')^ the function Ej{PxY,') has been characterized for all R. Specifically, it is shown in |j9l, 

ifTTl that 

E'}iPxY,R)=rRin[D{Qx\\Px)+E^iQx,PY\x,HiQx)-R)] ■ (24) 

W X 

Comparing (fT4l i with (fTSl l. (fTSl) with (fT9] l. (fThl l with (l20l) . and (fOl l with (l24l l. one can easily see that there exists 
an intimate connection between fixed-rate Slepian-Wolf coding for source distribution Pxy and channel coding for 
channel Py\x- This connection can be roughly interpreted as the manifestation of the following facts ifT^ . 

1) Given, for each type Qx G Pn{X), a constant composition code CniQx) Q TniQx) with R{Cn{Qx)) ~ 

H{Qx) - R and Pe.max(Cn(Qx), Py\x) ~ one can use CniQx) to partition type class T„iQx) 

into approximately disjoint subsets such that each subset is a constant composition code of type Qx 
with the maximum decoding error probability over channel Py\x approximately equal to or less than that 
of CniQx)- Note that these partitions, one for each type class, yield a fixed-rate Slepian-Wolf code of 
rate approximately R with Pr{X" ^ G XiQx)} ^ si^ce Pr{X'* G TniQx)} ~ 

^-nD(Qx\\Px) (cf. (ig, (O), it follows that Pr{X" ^ G TniQx)} ^ e-TD{Qx\\Px)+EiQx)]_ Xhe 

overall decoding error probability Pr{X" T 2f"} of the resulting Slepian-Wolf code can be upper-bounded, 
on the exponential scale, by e~T^^Tx\\Px)+E(Qx)\^ where Q\ = argming^ DiQx\\Px) + EiQx)- In 
contrast, one has the freedom to choose Qx in channel coding, which explains why maximization (instead 
of minimization) is used in (fT4l) . (fTSl) . and (fTSl l. 

2) Given a fixed-rate Slepian-Wolf code (t>ni') with Ri(f>n) ~ R and Pei4>n, Pxy) ~ e“"^, one can, for each 
type Qx G PniX), lift out a constant composition code CniQx) ^ TniQx) with RiCniQx)) ^ HiQx) — R 
and PeiCniQx),PY\x) ^ e-TE-D{.Qx\\Px)]_ 

3) The correct decoding exponents for channel coding and fixed-rate Slepian-Wolf coding can be interpreted in 
a similar way. Note that in channel coding, to maximize the correct decoding probability one has to minimize 
the correct decoding exponent; this is why in (fTSl) minimization (instead of maximization) is used. 

Therefore, it should be clear that to characterize the reliability functions for channel coding and fixed-rate Slepian- 
Wolf coding, it suffices to focus on constant composition codes. It will be shown in the next section that a similar 
reduction holds for variable-rate Slepian-Wolf coding. Indeed, the reliability function for constant component codes 
plays a predominant role in determining the fundamental rate-error tradeoff in variable-rate Slepian-Wolf coding. 
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III. Variable-Rate Slepian-Wole Coding: Above the Slepian-Wole Limit 


A variable-rate Slepian-Wolf code (pnO) is a mapping from A”” to a binary prefix code Bn- Let l{(j)n{x'^)) denote 


the length of binary string (pnix^). The ratj^ of variable-rate Slepian-Wolf code ipni,') is defined as 


R{^n,PxY) = -T^El{^n{X'-))- 
n log 2 e 

Given and L”, the output of the optimal maximum a posteriori (MAP) decoder is 

n 

X” = arg min -'^\ogPx\Y{x^\Y,) 

i—1 

n 

= arg min \ogPxY{xi,Y), 

X^-.Vn(X^)=V-n.{X^) ^ 

i—1 

where the ties are broken in an arbitrary manner. The decoding error probability of variable-rate Slepian-Wolf code 
Pn{') is defined as 


Pe{Pn,PxY) ^ X^}. 


The correct decoding probability of Slepian-Wolf code <pn{-) is defined as 

Pc{4’n, Pxy) = 1 — PeiP^n, Pxy) ■ 


Definition 4: Given a joint probability distribution Pxy, we say an error exponent i? > 0 is achievable with 
variable-rate Slepian-Wolf codes at rate R if for any (5 > 0, there exists a sequence of variable-rate Slepian-Wolf 
codes {pn} such that 

limsupi?(</5„,PxY) <RP5, 

n—>oo 

limsuplog Pe((/?„, Pxy) > E - S. 

n—xx) ^ 

The largest achievable error exponent at rate R is denoted by Ey{PxY , R)- The function Ey{PxY , ■) is referred 
to as the reliability function of variable-rate Slepian-Wolf coding. 

The power of variable-rate Slepian-Wolf coding results from its flexibility in rate allocation. Note that in fixed-rate 
Slepian-Wolf coding, one has to allocate the same amount of rate to each type clasj^. In general, the type that 
dominates the error probability of fixed-rate Slepian-Wolf coding is different from Px- In contrast, for variable-rate 
Slepian-Wolf coding, we can losslessly compress the sequences of types that are bounded away Px by allocating 
enough rate to those type classes (but its contribution to the overall rate is still negligible since the probability of 
those type classes are extremely small), and therefore, effectively eliminate the dominant error event in fixed-rate 


^It is worth noting that R{ipri, Pxy) depends on Pxy only through Px- 

^Since there are only polynomial number of types for any given n (cf. (T)), the encoder can convey the type information to the decoder using 
negligible amount of rate when n is large enough. Therefore, without loss of much generality, we can assume that the type of X" is known 
to the decoder. Under this assumption, an optimal fixed-rate Slepian-Wolf encoder of rate R should partition 71i(P) into min{|71i(P)|, e”^} 
disjoint subsets for each P G P„. It can be seen that the rate allocated to Tn{P) is always R if |Tii(P)| > e”^. 
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Slepian-Wolf coding. As a consequence, the types that can cause decoding error in variable-rate Slepian-Wolf coding 
must be very close to Px- This is the main intuition underlying the proof of the following theorem. 

Theorem 1: Ey{PxY,R) = E{Px,Py\x,H{Px) — R)- 

Proof: The proof is divided into two parts. Firstly, we shall show that Ey{PxY, R) > E{Px,Py\x, H (Px) — 
R). The main idea is that one can use a constant composition code of type approximately Px and rate 
approximately H{Px) — R to construct a variable-rate Slepian-Wolf code ipn'O) with n' ~ n, R{(pn', Pxy) ~ R, 
and Pe{ipn',PxY) < ^e,max (C„, TV|x)- 

By Dehnition|2] for any ^ > 0, there exists a sequence of constant composition codes {C„} with C 7^(P„) 
for some G Vn{X) such that 


lim ||P„ 

n—>oo 


= 0 , 


lim inf R{Cn) > H{Px) — R — 6, 

n—foo 

limsup -logPe,inax(Cn,TV|x) > E{Px , Py\x , H{Px) - R)-S. 

n—¥oo ^ 

Since Px{x) > 0 for all x G X, we have 


max 


PeVy.(x)ns{S) x p[x) 


Pn{x) ,, ,.,9 

max , < (1 + 


for all sufficiently n, where 

£{5) = \PG V{X) : max < 1 + <5, H{P) < H{Px) + <5, D{P\\Px) < 5 

y X P[x) 

Let kn = [(1 + (5)^n]. When n is large enough, we can, for each P G Vk„{X) fl £{S), construct a constant 
composition code C'^^{P) of length kn and type P by concatenating a hxed sequence in to each codeword 

in C„. It is easy to see that 


njP)\ = \Cn\, (25) 

Pe,max(Cfe„(P),PY|x) = Pe,ma.x{Cn, Py\x) (26) 


for all P G Pkn{X) n £{5). One can readily show by invoking the covering lemma in lfT3l that for each P G 
Pkn{X) n £{S), there exist L(/c„) permutations tti, • • • , TTL{k„) of the integers 1, • • • , such that 

L(k„) 

y McLiP)) = rkAP), 

2=1 


where 


L{kn) ^ max [|C;jP)|-i|rfc„(P)|log|rfc„(P)| + lJ. 

P^Vkn {P^)r\t{d) 

In view of (l25T l. we can rewrite L{kn) as 


^|7fen(f^)|l0g|Tfe„(P)| -f ij . 

F^Vkn 

Note that 


P. 


!:i'^ii^kniP))^ Py\x) — T’e,max(Cfe,^(P),PY|x): * 


= 1 , 2 ,-. 


, L{kn 


(27) 
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Given 7ri(C^jP)), • • 


It is clear that 


,T^L(kn){^kr^{P))’ partition Tk^{P) into L{kn) disjoint subsets; 

rfe„(p,i) = ^i(c^„(p)), 

rfe„(P,i) = 7r.(C;jP))\ lj7r.(4jP)), i = 2,..- ,L(fc„). 

G=i 


P. 


^{'Tk„{P,i),PY\x) < Pe,max(7ri(Cfc^(P)),Pr|Js:), i = 1,2, ••• ,L{kn). 


(28) 


Now construct a sequence of variable-rate Slepian-Wolf codes {<?!>fc„(-)} follows. 

1) The encoder sends the type of to the decoder, where each type is uniquely represented by a binary 
sequence of length 

2) If x^" G Tkn (P) for some P ^ £{6), the encoder sends losslessly to the decoder, where each x" G Tk^ (P) 
is uniquely represented by a binary sequence of length m 2 (kn)- 

3) If x^” G Tkn{P) for some P G £{5), the encoder finds the set ‘Ki*{C'^^{P)) that contains x*^" and sends the 
index i* to the decoder, where each index in {1, 2, • • • , L(fc„)} is uniquely represented by a binary sequence 
of length 7713 (fc„). 

Specifically, we choose 


TOi(fc„) = [log2 \VkAX)W, 

7712(fc„) = max [logs |7fe„(P)n, 

P&Vk^iX) 

mzikn) = [loga T(fc„)l. 


Note that 


where 


Riv’k,,, Pxy) 


m-i(fcn) + 9m2{kn) + (1 - 0)mz{kn) 

kn log 2 e 


Pv{X'^-GTkAP)}- 

PGPk„{x)nSiS) 

It is easy to verify (cf. ([T]l, (|2]i and @) that 

777l(fc„) < \X\ \ 0 g 2 ikn + 1) + 1, 

TO2(fc„) < fc„l0g2 \X\ + 1, 

+ 


Therefore, we have 


lim sup R{(l)k^, Pxy) 

n—^oo 


TO3(fc„) 

= lim sup -—^- 

n->oo kn l 0 g 2 6 

< max H{P) - ^ 

P€S{S) (l + < 5)2 


lim inf R{Cn 

n—¥oo 


< H{Px)+6- 


H{Px)-R-5 
(l + l5)2 ■ 


(29) 
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By (|2^ . (|27] |. (|28] | and the construction of it is clear that 

Pe{^k„,PxY) = Y. ^ Tfc„(P,i)}Pr{X’^ ^ e rfe„(P,i)} 

PG-Pfc„(Af)n£(5) i=l 

L{k„) 

^ E E Pr{X'=” e rfc„(P,i)}Pe,max(rfe„(P,i),Pr|x) 

PGPfc„(Af)ne(<5) i=i 

L{k„) 

^ E E Pr{X'=” G rfc„(P,i)}Pe.max(7r.(4„(P)),Pr|x) 

PGPfc„(x)ne(<5) i=i 

P(fc„) 

= E E Pr{x'=" G rfc„(P,i)}Pe.max(C„,Py|x) 

p^Vk^{x)ne{5) i=i 
E -ffijinax (Cn, Px|x), 


which implies 

limSUp--^logPe(<Pfe„,Pxy) > limsuplog Pe,max(C„, Py|x) 

n—^oo i^n n^oo 

^ E{Px.Py\x.H{Px)-R)-S 

( 1 + 5)2 • 

In view of (| 2 ^ . (l30l l and the fact that 5 > 0 is arbitrary, we must have Ev(Pxy, R) > E{Px, Py\Xt H{Px) — R) 
(cf. Definition nil. 

Now we proceed to show that Ey{PxY, R) < E{Px, Py\x,H{Px) — R)- The main idea is that one can extract 
a constant composition code of type approximately Px and rate approximately H{X) — R or greater from a given 
variable-rate Slepian-Wolf code +„,(•) of rate approximately R such that the average decoding error probability 
of this constant composition code over channel Py|x is bounded from above by j*Pe{(pn, Pxy), where 7 * is a 
constant that does not depend on n. 

By Definition m for any 5 > 0, there exists a sequence of variable-rate Slepian-Wolf codes {ipn} such that 


limsupP((p„, Pxy) < P + 5, 

(31) 

n—^oo 

limsup--logPe(+„,PxY) > Ey{PxY,R) -5. 

(32) 


n—¥oo P 


Suppose +«(■) induces apartitioqflof 7)i(P), P G P„(T’), into Nn{P) disjoint subsets 7)i(P, 1), • • • ,Tn{P, iVn(P)). 
Define 


P„(5) = |(P,z) : - log < P + 25, P G Vn{X), z = 1, 2, • • • ,W„(P)), 

I n |7;(P+)| J 

Qnin) = {(P,*) : Pr{X" ^ X”|X" G r„(P+)} < 7 Pe(+n,Pxy),P e Vn{X),i = 1 , 2 , - •• ,W„(P)} , 


^The pailition is defined as follows: (^„(a;") = <p„{x") if x'^,x‘^ G Tn(P,i) for some i, and + <p„{x’^) if x" G Tn{P,i),x^ G 

Tn{P,j) for i + j. 
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where 7 > 0. One can readily verify that 


^ Pr{X-e r„ (P,*)}>!- 




R{^n, Pxy) 
R + 25 ’ 


^ Pr{X"er„(P,*)}> 


7-1 


(P.*)ee„(7) 

Moreover, by OTI) and (l3^ we have 

liminf y Pr{X" G rn(P,i)} > 

n—>-oo ^ ^ 

(P.i)GP„(5) 

Let 7 * be a positive number satisfying 

7 * — 1 5 


R + 25 


7 * P + 2(5 


> 1 . 


Define 


P{x) 


Sn{5) = \P€ Vu{X) : H{P) > H{Px) - <5, max — < 1 + ^ ^ 

a; Px[x) 

D4^,7*) = {(P,i) : (P,i) G Fn{ 5 ) n gn{l*),P& 5„(5)}. 

It follows from the weak law of large numbers that 


lim y Pr{X” G r„(P)} = 1. 

n.—^ 


PG5„(5) 


In view of ( l?4] i. dLSl ) and ( |36] ). we have 


liminf y Pr{X" G rn(P,i)} 

rj .—^ 


(P,i)Gr>„(5,7*) 


> liminf < 1 — 

n—voo 


1- y Pr{X"Gr„(P,t)} 

(p.^)eP„(^) 


(33) 

(34) 

(35) 


(36) 


1- y Pr{X"Gr„(P,t)} 

(P7)eS„(7*) 


> 


^4 
7 * R + 25 


1 - y Pr{X"Gr„(P)} 

PdS+S) 

^ -1 


> 0 . 


Therefore, 'Dn{5,^*) is non-empty for all sufficiently large n. Pick an arbitrary {P*,i*) from I)ni5,'y*) for each 
sufficiently large n. We can construct a constant composition code of length m„ = [(1 + (5)n] and type Pm„ 
for some Pm„ G Pm„(‘L’) by concatenating a fixed sequence in to each sequence in %i{Pn+*) such that 

lim ||P™„-Pxll =0. (37) 

n—>-oo 
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Note that 

lim inf = 

n—foo 

> 

> 

Moreover, since 

= Pr{^" + ^"1^" € r„(P„^^*)} < 7*^^e(<P„,Pxy), 

it foiiows from (l32l i that 

iimsup- — log Pe{Cm„,PY\x) > ( 39 ) 

n^oo mn 1 + 0 

In view ofdSTll, dSI, ([391), and the fact that (5 > 0 is arbitrary, we must have Ey{PxY, R) + E{Px , Py\x t H {Px) — 

R) (cf. Definition |2|i. The proof is compiete. ■ 

The foiiowing resuit is an immediate consequence of Theorem [T] and Proposition 0] 

Corollary 1: Define 

Ev,ex{PxY, R) = Ef.x{Px 1 Py\X 1 H {Px) — R), 

Ev,rc{PxY,R) = Erc{Px , Py\X , H (Px) — R) , 

Ev,sp{PxY,R) = Esp{Px , Py\X , H {Px) — R)- 

We have 

1) Ey{PxY,R) > max{Ev^ex{PxY, R), Ey^rc{PxY, R)}', 

2) Ey(PxY, R) + Ey^sp{PxY, R) with the possibie exception of i? = H{Px) — R’^{Px, Py\x) which point 
the inequality not necessarily holds. 

Remark: 

1) We have Ey{PxY,R) = oo for R > H{Px) — R’^{Px , Py\x), and Ey{PxY,R) < oo for i? < H{Px) — 
R’^{Px,Py\x)- Therefore, H{Px) - R^{Px,Py\x) and H{Px) - R’^{Px,Py\x) are respectively the 
upper bound and the lower bound on the zero-error rate of variable-rate Slepian-Wolf coding. 

2) In view of (fTTT i. we have 

Ey{PxY,R) = Ey^sp{PxY,R) = Esp{Px , Py\x, H {Px) — R) 
for R G [H{X\Y), H{Px) - Rcr{Px, Py\x)]- Note that 

Ev^sp{PxY,R) > Ef^sp{PxY,R) > Ef{PxY,R), 

where the first inequality is strict unless the minimum in ( |20] | is achieved at Qx = Px, (i-e., Px(p) = Px, 
where Px(p) is the marginal distribution of induced by PyM and PxMiYip) (l2Tl l. (l22l l). There¬ 
fore, variable-rate Slepian-Wolf coding can outperform fixed-rate Slepian-Wolf coding in terms of rate-error 
tradeoff. 


lim inf-log |7;(P*, **) | 

n^oo rrin 


lim inf — 

n->-oo ?77rt 


■iog|r4/^:)l-i?-2<5 


H{Px) - R-3S 

1 + s ■ 


(38) 
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For R > H{Px) - Rcr{Px,PY\x)7 it is possible to obtain upper bounds on Ey{PxY, R) that are tighter than 
Ev,spiPxY , R)- Let Eex{PY\Xj R) Esp{Py\X} R) be respectively the expurgated exponent and the sphere 
packing exponent of channel Py\x- The straight-line exponent Esi{Py\X} R) of channel Py\x 121 is the smallest 
linear function of R which touches the curve Esp{Py\x, R) and also satisfies 

L'si(LV|X;0) = £'ea:(LV|Xi 0), 

where Eex{PY\XjO) L assumed to be finite. Let Rsi{Py\x) be the point at which Esi{Py\x, R) and Esp{Py\xj R) 
coincide. It is well known jS] that E{Py\X: R) < Esi{Py\x,R) for R G (0, Rsi(Pyix)]- Since E(Px,Pyix,R) < 
E(Pyix, R), it follows from Theorem [T] that 

Ex(Pxy,R) < E,i(Pyix,E(Px)-R) 

for R G [max{II(Px) - Rsi(Pyix), 0}, ff(Px))- 

Note that the straight-line exponent holds for arbitrary block codes; one can obtain further improvement at high 
rates by leveraging bounds tailored to constant composition codes. Let E*^(Qx,Pyix, 0) be the concave upper 
envelope of Eex(Qx, LV|Xi 0) considered as a function of Qx- In view of ||2] Excercise 5.21], we have 

E{Qx,Py\x,R) < e:^{Qx,Py\x,0) 

for any Qx GP{X) and i? > 0. Now it follows from Theorem [T] that 


Ex{Pxy.R)<E:^{Px,Py\x.Q) 


for R < H{Px). 

The following theorem provides the second order expansion of Ey{PxY, R) at the Slepian-Wolf limit. 

Theorem 2: Assuming Rcr{Px,Px\Y) < I{Px,Py\x) (see Proposition [T] for the necessary and sufficient 
condition), we have 


Ex{PxY,H{X\Y)+r) 

hm- t: - 

riO 


^ PxY{x,yy{x,y) - Y,Px{x) T{x,y)PY\x{y\x)\ 

x,y X \ y / 


where t(x, y) = logPy(y) - log Py|x(y|a;). 

Remark: If Pcr-(Px,Px|x) = /(Px,Px|x), then we have P«,rc(Pxx,P) = R- H{X\Y) for R > H{X\Y), 
which implies 


Ex{PxY,H{X\Y)Yr) 
hm-r-= oo. 

riO 

It is also worth noting that the second order expansion of Ey {Pxy , R) at the Slepian-Wolf limit yields the 
redundancy-error tradeoff constant of variable-rate Slepian-Wolf coding derived in iflTll . 
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Proof: Since R,r{Px,Px\Y) < /(Px; Py|Y), it follows that iJ(X|y)+r G iH{X\Y), H{Px)-Rcr{Px , Py\x)) 
when r (r > 0) is sufficiently close to zero. In this case, we have 

E,{PxY,H{X\Y)+r) Esp{Px,PY\x,IiPx,PY\x)-r) 


min 

Qy\x'HPx ,Qy\x)<HPx ,Py\x)-'^ 


min 

Qy\x-HPx ,Qy\x)=I{Px ,PY\x)-r 


D{Qy\x\\Py\x\Px) 

p2 

D{Qy\x\\Py\x\Px) 


where the last equality follows from the fact that Esp{Px, Py\x ^ R) is a strictly decreasing convex function of R 
for R G {Rf^{Px-,PY\x)',I{Px,PY\x)]- 

Let X{x,y) = QY\x{y\x) - PY\x{y\x) for x & X,y & y. Let A{y) = Y,^Px{x)A{x,y) for y G 3^. By the 
Taylor expansion, 

I{Px,Qy\x) = ^Px{x){PY\x{y\x) + A{x,y))\og{PY\x{y\x) + A{x,y)) 

x,y 

- '^{PYiy) + A{y)) \og{PY{y) + A{y)) 

y 

= ^Px{x){PY\x{y\x) + A{x,y)) (log PY\x{y\x) + + o{A{x,y)) 

- (^log-FV(t/) + + o(A(?/))^ 

= /(Px,Py|jf) -^(A(2/) + A(y)logPy(y) + o(Ay)) 
y 

+ ^Px{x){A{x,y) + A{x,y) log Py|x(y|a:) + o{A{x,y))) 


and 


P(Qy|xl|Py|Jc|Px) 

= S^’xWOv-i.vfaNiiogAj^ 


x,y 


= '^Px{x){PY\x{y\x) + A{x,y))\og (^1 + 

= E Px W(P.,xto|x) + A(x,,)) (+ o(A“(x,!,))) 

Here f{z) = o(z) means lim^-^o = 0. 

As r 4, 0, we have A(y) —>■ 0, A{x,y) —>■ 0 for all a; G A",?/ G y. Therefore, by ignoring the high order terms 
which do not affect the limit, we get 

P„(Pxy,P(A|y) + r) 


lim ■ 

r^O 


. ^ Px(x)A2(a;,t/) 

= limmm > - , , , ^ 

7^ 2Py|x(2/|a;)r2 
5 y 


(40) 
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where the minimization is over A{x,y) (x G X,y Gy) subject to the constraints 

1) J2y y) = 0 for all X G X; 

2 ) T,x,vPxix)T{x,y)A{x,y) =r. 

Introduce the Lagrange multipliers a{x) {x G X), (3 for these constraints, and define 

^ = ~ «( 2 ;) A(a;, y)- {x)t{x, y)A{x, y). 

x,y Y\X\y\ ) 

The Karush-Kuhn-Tucker conditions yield 

= -a{x) - pPx{x)T{x,y) + = 0, xGX,yGy. 


dA{x,y) 

Therefore, we have 


PY\x{y\x) 


Prixivlx) 


A{x,y) = l3T{x,y)PY\xiy\x) + a{x). 

Substituting fiTli into constraint 1), we obtain 

a{x) = -pPx (x) t(x, y)Prix (yjx) 
y 

which, together with (HTt , yields 

A(x,t/) = l3T{x,y)PY\x{y\x) - l3PY\x{y\x)'^T{x,y')PY\x{v'\x). 

y' 

Therefore, we have 

Px{x)Al 


(41) 


(42) 


E 


“^Prixivlx) 






-\ 2 


Tix, 2/) - E y')PY\xiy'\x) 


y' 


= -7G^PxY{x,y) 


\x, y) - 2t{x, y) t{x, y')PY\x{y'\x) + t{x, y')PY\x{.y'\x) 


E 

2 


y] Pxy{x, y)T^{x, y)-J2 PYx) T{x,y)PY\x{y\x) 


x,y 


(43) 


Constraint 2) and (l42li together yield 


^2 P- 


p- {'^Pxix)T{x,y)A(x,y) 


-\ 2 


'^Px{x)T{x,y) I T{x,y)PY\xiy\x) - PY\x{y\x)'^T{x,y')PY\x{y'\x) 

x,y \ y' 

'^PxY{x,y)T'^{x,y) -^Px{x) i^T{x,y)PY\x{y\x)\ 


(44) 


The proof is complete by substituting (|4^ and (l44li back into (l40l l. 
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IV. Variable-Rate Slepian-Wole Coding; Below the Slepian-Wolf Limit 

Definition 5: Given a joint probability distribution Pxy, we say a correct decoding exponent E'^ > 0 is achievable 
with variable-rate Slepian-Wolf codes at rate R if for any <5 > 0, there exists a sequence of variable-rate Slepian-Wolf 
codes {(fin} such that 

limsup R{(pn, Pxy) < R + S, 

n—¥C<D 

liminf-ilogPc(</5n,^’A:y) < E^ + 6. 

n—>-oo Tl 

The smallest achievable correct decoding exponent at rate R is denoted by E^{Pxy, R)- 

In view of Theorem[T] it is tempting to conjecture that E'^{Pxy, R) = E‘^{Px, Py\Xj H{Px) — R)- It turns out 
this is not true. We shall show that E^{Pxy , R) = 0 for all R. Actually we have a stronger result — the correct 
decoding probability of variable-rate Slepian-Wolf coding can be bounded away from zero even when R < H{X\Y). 
This is in sharp contrast with fixed-rate Slepian-Wolf coding for which the correct decoding probability decays to 
zero exponentially fast if the rate is below the Slepian-Wolf limit. To make the statement more precise, we need 
the following dehnition. 

Definition 6: Given a joint probability distribution Pxy, we say a correct decoding probability P(.^v{Pxy,R) is 
achievable with variable-rate Slepian-Wolf codes at rate R if for any J > 0, there exists a sequence of variable-rate 
Slepian-Wolf codes {ipn] such that 

\im sup R{(pn, Pxy) < R +6, 

n—^oo 

limsup Pci'fn, Pxy) > Pc,v{Pxy,R) - S. 

n—^cio 

The largest achievable correct decoding probability at rate R is denoted by P™^^{Pxy , R)- 
Theorem 3: P^^^{Pxy,R) = for R S {0,H{X\Y)]. 

Remark: It is obvious that Pfi)^^{PxY,R) = 1 for i? > H{X\Y). Moreover, since P™^^{Pxy, R) is a monotoni- 
cally increasing function of R, it follows that Pfi]^^{PxY,0) = 0. 

Proof: The intuition underlying the proof is as follows. Assume the rate is below the Slepian-Wolf limit, i.e., 
R < E[{X\Y). For each type P in the neighborhood of Px, the rate allocated to the type class %i{P) should be no 
less than H{X\Y) in order to correctly decode the sequences in Tn{P)- However, since almost all the probability 
are captured by the type classes whose types are in the neighborhood of Px, there is no enough rate to protect 
all of them. Note that if the rate is evenly allocated among these type classes, none of them can get enough rate; 
consequently, the correct decoding probability goes to zero. A good way is to protect only a portion of them to 
accumulate enough rate. Specihcally, we can protect jjf^x\Y) fraction of these type classes so that the rate allocated 
to each of them is about H{X\Y) and leave the remaining type classes unprotected. It turns out this strategy 
achieves the maximum correct decoding probability as the block length n goes to inhnity. Somewhat interestingly, 
although EI^j^Pxy, R) E‘^{Px, Py\x, H{Px) — R), the function E^{Px,Py\x, ') does play a fundamental role 
in establishing the correct result. 
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The proof is divided into two parts. Firstly, we shall show that P^^^{Pxy, R) > h{x\y) ■ e > 0, define 

U{e) = {PeP{X)-.\\P-Px\\<e}. 

Since Px{x) > 0 for all x G A", we can choose e small enough so that 


gmin(e) = min P{x) > 0. 
Pm(e),xeX 


Using Stirling’s approximation 


1 , /-- /my 

^—j < ml < V^irm ^—j 


we have, for any P €U{e)r\ Vn{X), 


Pr(X"Gr„(P)) = 


Uxi'^Qx{x))i 

y/2TTneT^ 




^ V^7rnei2n 

Ux V^TrnP{x) 

i-yi-i 

< - , n 2 

rix 

\f^e 12n lA-l-l 

< - , -n 2 ^ 

Ux \/27rgmin(e) 

which implies that Pr(X" G 7^(P)) converges uniformly to zero as n — >■ oo for all P Gl4{e)r\ Vn{X). Moreover, 
it follows from the weak law of large numbers that 


lim V Pr(X" G r„(P)) = 1 

n—>-oo • ^ 

P&J{e)f\Vn{^) 

Therefore, for any 5 > 0, P G {0, H{X\Y)], and sufficiently large n, we can find a set C U{e) nP„(A’) such 
that 

Now consider a sequence of variable-rate Slepian-Wolf codes {(/?„(•)} specified as follows. 

1) The encoder sends of type of X" to the decoder, where each type is uniquely represented by a binary sequence 
of length |■log 2 |P„(<T)n. 

2) For each P G 5„, the encoder partitions the type class 7^(P) into L„ subsets 7^(P, 1), Tn{P, 2), • • • ,Tn{,P, Ln). 
If X" G TniP) for some P G 5„, the encoder finds the subset Tn{Py*) that contains X" and sends the 
index i* to the decoder, where each index in {1, 2, • • • , P„} is uniquely represented by a binary sequence of 
length |■log 2 |P„n. 

3) The remaining type classes are left uncoded. 

Specifically, we let 
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It follows from |[3 Theorem 2] that for each P G Sn, it is possible to partition the type class %i{P) into L„ disjoint 
subsets 7^(P,l),7jt(P,2),-•• so that 

--logPr(X" 9 ^X"|X"Gr„(P))> min [Erc{Qx,PY\x,H{Qx) - H{X\Y) - 5) - e] 
n Qxew(e) 

uniformly for all P G Sn when n is sufficiently large. In view of the fact that Erc{Px ,Pyix,I(Px,Pyix)-S)>0 
and that Erc{Qx,PY\x,R) as a function of the pair {Qx,R) is uniformly equicontinuous, we have 


min [EreiQx,PY\x,H{Qx) - H{X\Y) - S) - e] ^ > 0 

Qx&U{e) 

for sufficiently small e. 

For this sequence of constructed variable-rate Slepian-wolf codes {ipn{-)}, it can be readily verified that 
limsup Pjvr) = limsup— - - 

n—)-oo n—^co ^ ^ 

and 


riog 2 \Pn{X)\^ + ^ Pr{X" G rniP)}Ln 

PGSr, 


limsupPc(‘Fn, > 

n—J-oo 

> 


> 



TniP)} 


H{X\Y) 


Since i5 > 0 is arbitrary, it follows from Definition |6] that P™^^{Pxy,R) > h{x\y) ■ 

Now we proceed to prove the other direction. It follows from Definition |6] that for any (5 > 0, there exists a 


sequence of variable-rate Slepian-Wolf codes {</2n(-)} 


limsupi?(i^„,PxY) < R + S, 

n—foo 

limsupPc(‘Fn: Pxy) > P^y'^iR) - S. 

n—¥oo 

Define 


r{Tn{P)) = 


1 


\Tn[P)\n\og2e 


^ P&Pn{X). 


X^&TXP) 

Since R{ipn, Pxy) = SpeP ^ 'Tn{P)}'r{Tn{P)), we can interpret r{Tn{P)) as the rate allocated to 

the type class Tn{P)- 

For each P G U{e) fl P„(T’), suppose partitions the type class 7j^(P) into N{P) disjoint subsets 

Tn{P,l),--- ,Pn{P,N{P)) (i.e., (finix"-) = (pn{x^) if G Tn{P,i) for some i, and ^ (finix'^) 

if a;" G Tn{P,i),x'^ G TniPJ) for i ^ j). Define 

In{P,5) = \i-.- log <H{X\Y)-5,i = l,2,-- - , N{P) 


n \Tn{P,i)\ 

X;;(P,J) = ^:ilog^^>iJ(X|y)-J,* = l, 2 ,... , 1 V(P)!>. 
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Note that 


which implies 


rITiP)) > |2,(P)| 

' - nh \Tn{P}\ '|r.(-P..)l 


> 


1 ^ irn(p,i)| |r„(p)| 

^exfTp.) 


> (if(x|r)-5) ^ 


iei‘iP,s) 


ir„(p,i)| 
ir„(p)| ’ 


^ |rn(p,»)| ^ r(r„(p)) 

^ |r„(P)| - H{X\Y)-6' 


ieI„(P,5) 

Each Tn{P,i) can be viewed as a constant composition code of type P and we have 


Pr{X” = X"|X" G Tn{P,i)} = Pe(r„(P,2),Py|x)- 


Note that for P ^U{t) C\'Pn{X) and i G Tn{P, <5), 


iiog|r„(p,i)| > -iog|r„(p)|-p(x|y) + 5 

n n 


> H{P)-H{X\Y) + 5-\X\ 


log(n + 1) 


Therefore, it follows from IH Lemma 5] that 

-ilogP,(r„(P,i),Px|x) > min E<^{Qx,Py\x,H{Qx) - H{X\Y) + 5- e) - e 
n Qxew(£) 

uniformly for all P G W(e) fl P„(T’) and i G In{P,5) when n is sufficiently large. In view of the fact that 
P°(Px, Py|X)-^(Px, Pf|x) + 5) > 0 and that E'^{Qx,Py\x,R) as a function of the pair {Qx,R) is uniformly 
equicontinuous, we have 

min [E^iQx,PY\x,H{Qx) - H{X\Y) + S - e) - e] ^ K 2 > 0 

Qx&U{t) 

for sufficiently small e. 

Now it is easy to see that 


liminf Pe((/?„, Pxv) 

n—)-oo 


> 


liminf ^ Pr{X"Gr„(P)} ^ (l - Pr{X-= G r„(P, i 

n^oo ^ ^ \Tn[P)\ ^ 


PGW(e)nPn(X) 


ieI„(P.<5) 


> liminf y Pr{X"Gr„(P)} V ^^^(1 - 

rj,—foo X—£ Z—£ I” ' 


PGW(£)nP„(X) 


ieI„(P,(5) 


ir„(p)| 


> liminf y Pr{X” G Tn{P)} (1 - 

n.— r ^ \ 


PGW(£)np„(x) 


rjTnjP)) 

H{X\Y)-S 


(l_g-««2) 


> liminf y Pr{X" G 7;(P)}(1 - 

rj.— r ^ 


PG2^(e)n'Pn(Af) 
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-limsup ^ Pr{X" g Tn{P)} ^ 4 1 - e—) 

™ HiX\Y)-d 

> liminf V Pr{X” G 7;(P)}(1 - e"”'"") 

n—¥oo • ^ 

Pei^{e)n'Pn{A') 

-limsup ^ Pr{X" G TniP)} ^ 4 1 - e-"--) 

= liminf V Pr{X” G 7;(P)}(1 - 

n—>-oo ^ ^ 

Pei^(e)nPn(A’) 

Ri^Pn, Pxy) 


— lim sup 
= 1 - 


„^oo H{X\Y)-5 
i? + (5 


(l_g-««2) 


i/(x|r)-5’ 


which implies 


limsup Pc(V3n,^xy) < 


R + 5 


Therefore, we have 


P“r(Pxy,P)-5< 


H{X\Y) -5' 
R + 5 

H{X\Y)-5' 


Since 5 > 0 is arbitrary, this completes the proof. 


V. Example 

Consider the joint distribution Pxy over Z 2 x Z 2 with Px\ y(l|0) = Px |y(0|l) = p and Py(0) = r. We assume 
p G (0, ^), T G (0, i]. It is easy to compute that 


Px(0) = 1 - Px(l) = r(l -p) + (1 - t)p, 

Py\xm = 1 - Py|x(0|0) = 

P^|^(0|1) = 1 - Py|^(l|l) = 

For this joint distribution, we have H{X\Y) = Ht{p), where P{,(-) is the binary entropy function (i.e., Hb{p) = 
—plogp — (1 — p) log(l — p)). Given R G [0, log2], let q be the unique number satisfying Hi,{q) = R and q < 

It can be verified that 


Ef^sp{PxY,R) = D{q\\p), R G [P&(p),log2], 
E%Pxy,R) = D{q\\p), R G [0,Pfc(p)]. 


Note that 


Eex{Qx,PY\X,^) = — Qx{X)Qx{x') log 


XI i/^x|x(p|a:)Py|jf(j/|a;') 


= —2Qx(0)Qx(l) log 


X \/-fV|x(p|0)Py|x(p|l) 
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which is a concave function of Qx- Therefore, 

ElxiPx,PY\X, 0 ) = Eex{Px: Py\X, 0 ). 

Moreover, we have 


Eex{PY\XfO) 


It is easy to show that 


maxEex{Qx,PY\x,0) 

Qx 


-llog 


\/^l.\'(y|0)Py|x(y|i) 


L y 


Ev,sp{PxY, H{Px)) 


Esp{Px,Px\Y: 0) 


Px(a;) 

X y 


Qyjy) 

PY\xiy\x) 


where the minimizer Qy is given by 


Q*Y{y) 


Ux PY\xiy\x)^^^^^ 

Ey'UxPYixiy'lxr-^^P 


Define 


Ef^eriPxY, R) = n[iax{Ef^ex{PxY , R), E f ^rc{PxY , R)}, 

Ev,er{PxY^ R) = max{P v^ex {PxY,R), Ey ,rc {Pxy,R)}- 

We have 


Ef{PxY,R) > Ef^er{PxY, R), 

Ev{PxY , R) > Ey^er{PxY,R)- 

It can be seen from Fig.|2]that the achievable error exponent Ey^er{PxY, R) of variable-rate Slepian-Wolf coding 
can completely dominate the sphere packing exponent Ef^sp{PxY,R) of fixed-rate Slepian-Wolf coding. The gain 
of variable-rate coding gradually diminishes as r —i (see Fig. |3]and Fig. HJl. 


VI. Concluding Remarks 

We have studied the reliability function of variable-rate Slepian-Wolf coding. An intimate connection between 
variable-rate Slepian-Wolf codes and constant composition codes has been revealed. It is shown that variable-rate 
Slepian-Wolf coding can outperform fixed-rate Slepian-Wolf coding in terms of rate-error tradeoff. Finally, we 
would like to mention that Theorem [T] has been generalized by Weinberger and Merhav in their recent paper on 
the optimal tradeoff between the error exponent and the excess-rate exponent of variable-rate Slepian-Wolf coding 

Ea. 
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R 

Fig. 2. p = 0.05, T = 0.12 



R 

Fig. 3. p = 0.05, T = 0.35 
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Fig. 4. p = 0.05, T = 0.50 

Appendix A 

Proof of Proposition[T| 

In view of (|7|i and (fTTIl . we have Rcr{Qx, Wy\x) = HQx, Wy\x) if and only if the minimum of the convex 
optimization problem 


min D{Vyix\\Wyix\Qx) + /(Qx, fV|x) 

|x 


(45) 


is achieved at Vy\x = Wy\x- Let Vy\^x ^ minimizer to the above optimization problem. Note that for a;, y such 
that Qx{x)WY\x{y\x) = 0, there is no loss of generality in setting VY\x{y\^) — WY\x{y\x)- Let A = {x € X : 
Qx{x) > 0} and Bx = {y & y ■ WY\x{y\x) > 0} for x € A. We can rewrite (l45l l in the following equivalent 
form: 


min 

VY\x(y\A'-^^Xv^B, 


^ Qx{x)VY\x{y\x)\og 



xGA,yGBx 


WY\x{y\x) Y^x'ga Qx{x')VY\x{y\x') 


subject to 


VY\x{y\x)>Q fox &\\ X € A,y € Bx, 
VY\x{y\x) = 1 for all x & A. 
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Define 


G' 


^ Qxix)VY\xiy\x)\og 

xGA,y^Bx 


Vylxiyl^) 

WY\x{y\x) Y^x'dA Qx{x')VYix{y\x') 


- Pix)VY\x{y\x), 

x^A.y^Bx x£A,y£Bx 


where a{x,y) G ffi.+ (x G A,y G Bx) and /3(x) S K. (x S A). The Karush-Kuhn-Tucker conditions yield 

= 2Qx{x* ) log V^\x{y*\x*) + Qx{x*) - Qx(x* ) log Wyixiy* I) 

tV|x(y*k*)=Vjt|x(y*k*) 

-Qx{x*)log ^ Qx{x')VYix{y*\x') - oi{x*,y*) - ^{x*) 
x'£A 

= 0 for all X* G Aj/* e 



V^ixiy*\^*)>0 forallx* G Ay* eB,., 

^r|x(y*k*) = l forallx*GA 

y*eBx* 

a{x\y*)V*^xiy*\x*) = 0 for all x* G Ay* e -B,.. 

By the complementary slackness conditions (i.e., A'|^(?/*|x*) > 0 a{x*,y*) = 0), we have ~ ^f| 2 s: if 

and only if for all x* G A y* G Bx*, 

Qx(a;*)logVlA|x(y*|A) + Qxix*) - Qx{x*)\og ^ Qx{x')WY\xiy*\x') - /3(A) = 0, 

x'^A 

i.e., the value of 

, WAix(ylA 

Ex' Qx{x')WY\x{y\x') 

does not depend on y for all x,y such that Q 2 s:(x)WV| 2 s:(y|a;) > 0. 


Appendix B 

Proof of Proposition^ 

1) It is known ^ Exercise 5.17] that for every i? > 0, <5 > 0 and every P G Vni^) there exists a constant 
composition code A C Tn{P) such that 

R{Cn) >R-S, 

-logPe,max(C„, > Eex{P,WY\X, R) — 5 

n 

whenever n > no(A|, A|,^). Let be a sequence of types with Pn G Pn{A) and 


Define 


lim IIP„ - Q^ll = 0. 


V* = argmin 


^ ^ (a^)Tn (x|x)d^y (x, x) T 7(Tn; fn) R 

X,X 
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where the minimization is over Vn '■ X ^ X subject to the constraints 

Pn{x)Vnmx) = Pnix), for all X G X, 

X 

IiPn,Vn)<R. 

Note that {V*} must contain a converging subsequence {V*^}. Define 


V* = lim V*. 

71—^00 ^ 


It is easy to verify that 


J2Qx{x)V*{x\x) = lim (xlx) 


xga: 


x€a: 


Therefore, we have 


= lim P„(x) 

k—>oc 

= Qx(x), for all a? G A”, 
I(Qx,V*) = lim/(P„,,Krj 

/c—>oo 

< E. 


lim sup Eex (Pn ,Wy\X, R) 

n—¥oo 

> \im sup Eex{Pnk,WY\X,R) 

k—¥co 

= lim sup E Prik {x)Xj^^ {x\x)d\YYix (^5 X I{Pnk ; ) R 


k—¥oo 




> 


Tl Qxix)V*{x\x)dpY^^{x,x) + I{Qx, V*) - R 

x,x^X 

> Eex{Qx,WY\XTR)- 

It is also known ^ Theorem 5.2] that for every R > 0, 6 > 0 and every P G Vn{X) there exists a constant 
composition code C„ C Tn{P) such that 

R{Cn) >R-5, 

-log Pe,niax{Cn, Wy\x) > Erc{P, Wy\X, R) — S 

n 

whenever n > no(|T’|, |3^|,5). So it can be readily shown that 


E{Qx,Wy\XtR) > Erc{Qx,WY\XTR) 

by invoking the fact that Erc{P,WY\x,R) as a function of the pair {P,R) is uniformly equicontinuous ||2l 
Lemma 5.5]. The proof is complete. 
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2) By Definition m for every R > 0, S > 0 there exists a sequence of block channel codes codes {C„} with 
Cn Q Tn{Pn) for some Pn G Vn{^) such that 

lim ||P„ -QxW = 0, 

n—>oo 

lim inf R{Cn) > R — S, 

n—^oo 


limsup- logPe.max(C„, fkVlJt) > E{Qx,Wy\X,R) - S- 


(46) 


(47) 


min D{Vy\x\\Wy\x\Qx)- 

Vyix-HQx,Vyix)<R-35 


For simplicity, we assume R{Cn) > R — 6 for all n. Now it follows from Theorem 5.3 in [8] that 
- i log [2Pe,^ax(C„, Wyix)] < P.p(P„, , -R - 25)(1 + <5) 

whenever n > no(|<T|, |3^|,5). Let 
V^\x = arg_ 

Without loss of generality, we can set = Wy\x{'\E) for all x S {x' G X : Qx{x') = 0}. It is easy 

to see that there exists an e > 0 such that 

/(P, < P - 2<5, 

D{V^\xWy\x\P) < D{Vi^x\\WY\x\Qx) + <5 

for all P G P(T’) with ||P — Qx\\ < £■ Therefore, for all sufficiently large n, 

P.p(Pn,VLy|x,P-2<5) = min D{VY\x\\WY\x\Pn) 

VVIX :(-Pn , Vy IX ) <-R - 3 (5 

< D{V^^^\\WY\x\Pn) 

< D{V^^^\\Wy\x\Qx)+5 

= P.p(Qx,ILy|x,P-3<5) + L (48) 

Combining ( |46] |, ( l47l i and ( |48] ). we get 

E{Qx, Wy\x.R) - 5 < [P.p(Qx, Wy\x,R - 3<5) + 5](1 + <5). 

In view of the fact that (5 > 0 is arbitrary and that for fixed P and Wy\x 7 Esp{P, ITV|x, P) is a decreasing 
continuous convex function of P in the interval where it is hnite ^ Lemma 5.4], the proof is complete. 

3) It is known a Lemma 5] that for every P > 0, <5 > 0, every constant composition code of common type 
P for some P G Vn{X) and rate R(Cn) > R + S has 

-logPc(C„, Wy\x) P min \_D{Vy\x\\Wy\x\P) + |P — 7(P, fV|x)|^] — <5 

n Vy\x 

whenever n > no(|<T|, |3^|,(5). Moreover, it is also known 0 Lemma 2] a Excercise 5.16] that for every 
R > 0, 5 > 0 and every P G Pn{X) there exists a constant composition code Cn C Tn{P) such that 

P(C„) > P - <5, 

-logPc(C„, tTV|x) > ruin [D{Vy\x\\'^y\x\P) + |P — I{P, I4'|x)|'''] + ^ 

n vVjx 
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whenever n < no(|A’|, |3^|, <5). In view of the fact that minvy|^ [D{Vy\x\\Wy\x\P) + |-R — f(-P, VV|x)|'''] 
as a function of the pair (P, R) is uniformly equicontinuous, it can be readily shown that 

Ec{Qx,Wy\x,R) = min [D{Vy\x\\Wy\x\Qx) + |P — I[P, 14"|jf)|^] • 

Vy\X 

The proof is complete. 
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